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METHOD AND APPARATUS FOR REMOVING 
DEFECTS IN AN IMAGE SEQUENCE 

Field of the Invention 

5 The present invention relates to digital image 

restoration techniques, and more particularly, to a method and 
apparatus for removing defects from images. 

Background of the Invention 

10 Digital motion picture restoration involves a variety 

of image processing operations that require a considerable amount 
of computations and take a relatively long amount of time, even 
with powerful computers. An example of a computation-intensive 
image processing technique is motion estimation, which is 
*pl5 required for high fidelity repair. Generally, motion estimation 
CO allows a defective region of an image to be replaced with a 
m corresponding image region from a previous or subsequent image 
D frame . 

(*% Longer processing times are acceptable if the 

=20 computations are conducted automatically, without user 
' s tl interaction. Numerous complex algorithms have been developed for 
automatic image sequence restoration. For a detailed discussion 

. 

%~* 

p of such image restoration techniques, see, for example, Anil 
' fsh Kokaram, "Motion Picture Restoration," (Springer, 1998). 
25 Currently available image restoration techniques, however, do not 
provide a complete solution. Generally, currently available 
image restoration techniques suffer from incomplete repair or 
false detection (or both), resulting in image artifacts. Thus, 
human intervention is eventually required to complete the image 
30 restoration and obtain high quality artifact-free restoration. 

FIG. 1 illustrates a conventional one-step image 
restoration process 100, where an operator 110 works with an 
unprocessed, original image sequence 120 to produce a repaired 
image sequence 140. The image restoration process shown in FIG. 
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1 has been proposed, for example, for use in the image 
restoration software and services offered by Mathematical 
Technologies, Inc. of Providence, RI . Generally, the operations 
performed during an interactive session 130 are conducted under 
5 the supervision of the operator 110 who manually identifies 
defects in the original image sequence 120. Thus, the image 
restoration process 100 does not make efficient use of expensive 
operator time. Once defects are identified by the operator in 
this manner, well-known restoration algorithms are typically 
10 applied during the interactive session 130 to generate the 
repaired image sequence 140. 

FIG. 2 illustrates a partially automated image 
restoration process 200. As shown in FIG. 2, the operator 210 
7l initially specifies parameters 215 that control the automatic 
^15 detection and repair of defects in the original image sequence 
U 220 during an automatic repair stage 230. The operator 210 can 
■j review the repaired image sequence 240 generated by the automatic 
^ repair stage 230 together with the original image sequence 220, 

13? 

and accept, reject or modify any of the automatic repairs during 
^0 an interactive session 250 to produce a repaired image sequence 
L 260. An example of the partially automated image restoration 
~ process 200 shown in FIG. 2 is the Revival™ software package 
^ commercially available from da Vinci Systems, Inc. of Fort 

Lauderdale, FL. 

25 While the partially automated image restoration process 

200 automatically detects and repairs images without human 
interaction in a more efficient manner than the one-step image 
restoration process 100 of FIG. 1, the operator 210 still has 
very limited abilities to mark and repair additional defects that 

30 are not automatically detected during the automatic repair 230. 
Thus, it is difficult for the operator 210 to achieve a high 
quality image restoration during the interactive session 250 
because the intermediate results from the automatic repair 
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processing 230, such as motion estimation and granularity 
information, are not available. A need therefore exists for a 
method and apparatus for removing defects from images that 
improve the quality of the image repair while also making 
5 efficient use of the operator's time. 

Summary of the Invention 

Generally, a method and apparatus are disclosed for 
detecting and removing defects from images to more efficiently 
10 generate repaired images with improved repair quality. An 
original image sequence is initially processed using conventional 
techniques to automatically identify and optionally repair 
defects. The present invention extends the conventional image 
w processing techniques by generating non-image data containing 
{$\15 intermediate results from the automatic detection and repair of 

J;f defects. In this manner, the non-image data may be employed in a 

HI 

□ subsequent interactive session to more efficiently repair defects 

f Jll identified by a user. 

2 The non-image data includes, for example, the motion 

-:f20 estimation, granularity and defect size/location information that 

" t 

U was utilized by the conventional techniques to initially identify 
:^ any defects. The present invention allows the operator to review 
M the original and repaired image sequences, and optionally the 
non-image data, in an interactive session to accept, reject or 
25 further modify any of the automatic repairs in a more efficient 
manner. Once . defects are identified by a user during an 
interactive session, the non-image data can be employed to more 
efficiently identify a replacement sub-image for the identified 
defective region. 

30 A more complete understanding of the present invention, 

as well as further features and advantages of the present 
invention, will be obtained by reference to the following 
detailed description and drawings. 
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Brief Description of the Drawings 

FIG. 1 is a flow chart describing a conventional one- 
step image restoration process; 
5 FIG. 2 is a flow chart describing a conventional 

partially automated image restoration process; 

FIG. 3 is a flow chart describing an image restoration 
process in accordance with the present invention; 

FIG. 4 illustrates an image defect removal system in 
10 accordance with the present invention; 

FIGS. 5A and 5B, collectively, are a flow chart 
describing an exemplary intermediate image analysis process 
embodying principles of the present invention; and 
lf\ FIG. 6 is a flow chart ' describing an exemplary 

KI15 interactive image repair process embodying principles of the 

5 p § 

m present invention, 
p Detailed Description 

:L FIG. 3 illustrates an image restoration process 300 in 

J;!20 accordance with the present invention. As shown in FIG. 3, the 
\2 operator 310 can optionally specify parameters 315 that control 
□ the automatic detection and repair of defects in the original 
image sequence 320 in a similar manner to the partially automated 
image restoration process shown in FIG. 2. The parameters 315 
25 may be specified by the user or default values may be employed. 
The parameters 315 may include, e.g., variables for thresholds, 
smoothness and precision. The original image sequence 320 is 
processed during an analysis stage 500, discussed below in 
conjunction with FIGS. 5A and 5B, that automatically identifies 
30 and repairs defects in a well-known manner to generate an 
automatically repaired image sequence 340. 

According to a feature of the present invention, the 
analysis stage 500 also generates non-image data 350 containing 
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intermediate results from the automatic detection and repair of 
defects during step 500. The non-image data 350 includes, for 
example, the motion estimation and granularity information 
utilized to repair any defects. In this manner, the operator 310 
5 can review the repaired image sequence 340, as well as the non- 
image data 350, together with the original image sequence 320, in 
an interactive session 600, discussed below in conjunction with 
FIG. 6, to accept, reject or modify any of the automatic repairs. 
As discussed more fully below, the non-image data 350 allows the 
10 operator 310 to control the interactive session 600 in a more 
efficient manner to produce a repaired image sequence 360. 

FIG. 4 illustrates an image defect removal system 400 
in accordance with the present invention. As shown in FIG. 4, 
y the image defect removal system 4 00 includes certain standard 
[iQlS hardware components, such as a processor 410 and a data storage 
'ff device 420, such as a read-only memory and/or a random access 
□ memory (RAM) . 

%l The data storage device 420 includes an intermediate 

= image analysis process 500 and an interactive image repair 

"p"20 process 600, discussed further below in conjunction with FIGS. 5 
M and 6, respectively. Generally, the intermediate image analysis 

* I ; 

process 500 identifies and optionally repairs defects to generate 
M an automatically repaired image sequence 340 and the non-image 
data 350 containing the intermediate results. The interactive 

25 image repair process 600 allows the operator to review the 
repaired image sequence 340 and original image sequence 320, as 
well as the non-image data 350 created by the intermediate image 
analysis process 500. The operator can mark certain areas of an 
image as defective and request further repair using the non-image 

30 data 350 in accordance with the present invention. 

The methods and apparatus discussed herein may be 
distributed as an article of manufacture that itself comprises a 
computer readable medium having computer readable code means 
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embodied thereon. The computer readable program code means is 
operable, in conjunction with a computer system, to carry out all 
or some of the steps to perform the methods or create the 
apparatuses discussed herein. The computer readable medium may be 
5 a recordable medium (e.g., floppy disks, hard drives, compact 
disks, or memory cards) or may be a transmission medium (e.g., a 
network comprising fiber-optics, the world-wide web, cables, or a 
wireless channel using time-division multiple access, code- 
division multiple access, or other radio-frequency channel) . Any 
10 medium known or developed that can store information suitable for 
use with a computer system may be used. The computer-readable 
code means is any mechanism for allowing a computer to read 
instructions and data, such as magnetic variations on a magnetic 
™* media or height variations on the surface of a compact disk. 
flj 15 Data storage device 420 will configure the processor 

fl 410 to implement the methods, steps, and functions disclosed 
□ herein. The data storage device 420 could be distributed or local 

L. s and the processor 410 could be distributed or singular. The data 

Q 

s storage device 420 could be implemented as an electrical, 

^f20 magnetic or optical memory, or any combination of these or other 
types of storage devices. It should be noted that, although data 
!jr: storage device 420 is shown separately from other elements of the 
U image defect removal system 400, this is not necessarily the case 
for all applications. Moreover, the term "memory" or "data 
25 storage device" should be construed broadly enough to encompass 
any information able to be read from or written to an address in 
the addressable space accessed by processor 410. With this 
definition, information on a network is still within data storage 
device 420 of the image defect removal system 400 because the 
30 processor 410 can retrieve the information from the network. 
In a further variation, the image defect removal system 400 may 
be embodied as an application specific integrated circuit (ASIC) 
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that may be incorporated, for example, in a workstation or other 
image processing device. 

FIGS. 5A and 5B, collectively, are a flow chart 
describing an exemplary intermediate image analysis process 500 
5 embodying principles of the present invention. As previously 
indicated, the intermediate image analysis process 500 identifies 
and optionally repairs defects to generate an automatically 
repaired image sequence 340 and also generates the non-image data 
350 containing the intermediate results. As shown in FIG. 5, the 
10 intermediate image analysis process 500 initially obtains the 
original image sequence 320 during step 510. Thereafter, the 
intermediate image analysis process 500 associates each image 
frame with a given scene during step 520. The image granularity 
^ is computed during step 530. 

fgl5 Steady defects in a sequence of images, such as 

vertical scratches, are identified during step 540. Steady 

—EL 3 

r| defects are those defects that are identified without the aid of 
j!** motion estimation information. For a detailed discussion of 
3 suitable defect detection techniques, see, for example, Anil 

% *?20 Kokaram, "Motion Picture Restoration," (Springer, 1998), 
"H> incorporated by reference herein. Thereafter, motion estimation 

13 S. 

s. i 5 

«: displacement vectors are computed during step 545 for each pixel 
H (or group of pixels) to previous and subsequent frames. The 
computed motion estimation displacement vectors are stored in the 
25 non-image data file 350 during step 550 and the image granularity 
data from step 530 is stored in the non-image data file 350 
during step 555. 

Any temporal discontinuities not explained by the 
motion vectors are identified as potential defects during step 
30 560 (FIG. 5B) . Morphological processing techniques are employed 
during step 565 to merge potential defects. A binary file is 
created during step 570 as part of the non-image data 350 
indicating the location and shape of each potential defect. 
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Each remaining potential defect is analyzed during step 
575 for contrast level deviation, e.g., relative to a simple 
interpolation, to determine if it is a real defect. For each 
real defect identified during step 575, the defect is replaced 
during step 585 with a corresponding region of a previous or 
subsequent image using the motion estimation information. For a 
detailed discussion of suitable defect repair techniques, see, 
for example, Anil Kokaram, "Motion Picture Restoration," 
(Springer, 1998), incorporated by reference herein. Thereafter, 
program control terminates. 

FIG. 6 is a flow chart describing an exemplary 
interactive image repair process 600 embodying principles of the 
present invention. As previously indicated, the interactive 
image repair process 600 allows the operator to mark certain 
areas of an image as defective and request further repair or 
modification using the non-image data 350 in accordance with the 
present invention. As shown in FIG. 6, the interactive image 
repair process 600 initially receives a user selection during 
step '610 of an image area for further modification and an 
indication of the type of modification to be performed, such as a 
general repair. 

Thereafter, the interactive image repair process 600 
identifies a replacement image for the user selected defect area 
during step 620, using, e.g., the granularity or motion 
estimation information (or both) from the non-image data 350. 
The user selected defect area is replaced during step 630 with 
the automatically identified replacement sub-image during step 
630, before program control terminates (or returns to step 610 to 
wait for another user input) . 

It is to be understood that the embodiments and 
variations shown and described herein are merely illustrative of 
the principles of this invention and that various modifications 
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may be implemented by those skilled in the art without departing 
from the scope and spirit of the invention. 



